Assistive technology (AT) products exist to help computer users who have a need for assistance in areas of learning, communication and access to information contained in and presented by computer software. These products have a need for information relevant to the computer interface. Similarly, existing automated testing products and user interface commanding utilities also have a need for information about the user interface. Currently, these products have no sufficient source of user interface (UI) information. These three types of products (clients) are required to have necessary support from elsewhere to enable them to: (1) gather information about an application's user interface; (2) programmatically discover and interrogate UI elements regardless of the technology used to build the UI; (3) generate keyboard and pointer input; and (4) understand what type of behavior or functionality is currently available. No single technology is available currently that gives an AT product all of these capabilities. Furthermore, current AT products are not always compatible with all graphical operating system (OS) technologies and lack the ability to filter and coordinate redundant or misleading notifications in a centralized way. An additional disadvantage is that current automation and accessibility infrastructures are not extensible and therefore require OS level changes to add new functionality.
Furthermore, currently to gather information about an application's user interface, the AT product must write application-specific code to obtain information for the user. The process of writing this application-specific code is time consuming and requires continuous maintenance. Current automation infrastructure also lacks the ability to filter and coordinate redundant or misleading event notifications in a consistent manner. Thus, event consumers are required to independently filter information.
Current systems allow AT products to request event notifications in three levels of granularity: (1) everything on a desktop; (2) in a particular process (such as opening of a word processor); or (3) in a thread in the particular process (multiple objects doing work in the process). Currently, when the client receives an event, it receives a window handle for a specific window in which the event occurred and other bits of information to indicate where the event occurred. A client can make a cross process call to retrieve the UI object that is related to the event. With this object, the client can make additional cross process calls to ask for information about that object. If the client needs five pieces of information, then the client must make five cross process calls. Cross process calls are exceedingly slow, so the performance cost of collecting UI information using current accessibility infrastructure is high. This type of known scenario is shown in FIG. 8. A server application 12 fires an event 6. A kernel 14 determines which clients must be notified and sends an event notification 18 to an interested client 10. The client 10 makes a request 16 from the server application 12 across a process boundary 2 for the object related to the event notification 18. The server application 12 returns the object 20 and then the client 10 can begin sending requests 16 for information about the UI control that fired the event. The server application 12 returns the requested information 20 across the process boundary 2 to the client 10.
Another current option allows client code to be loaded as a dynamic link library (.DLL) within a process. This option has several drawbacks. First, it requires cooperation from the system to load client code into a process. Second, it gives rise to security issues because once in the client code is loaded into an application's process, it is difficult to restrict the information it gathers. Third, to be an effective technique for the client, it must be loaded into every process on the system. Optimally, only trusted clients should be loaded into another application's process.
Furthermore, a system is needed that gives the client the ability to specify what event notifications it wants to receive. In known systems, a client may need to make a number of cross process calls and then analyze the information to determine if it is interested in the event. A mechanism is needed that can perform this event filtering in a more performant manner and that can be easily updated to support new system or application events. Furthermore, a system is needed that uses only trusted components in order to alleviate security concerns.
Currently, when seeking information about a user interface, the AT product is required to access trees that are native to a particular UI framework. Accordingly, multiple trees are required to convey user interface information for multiple UI frameworks. These differing trees may contain information that is not of interest or is not visible to the user, such as hidden container objects that manage the visible UI controls manipulated by the end user. Therefore, a need exists for a single unified tree having only those nodes that are of interest to the user.
A solution is needed that addresses the needs of AT products, automated testing tools, and commanding utilities. The solution should be usable by all graphical OS technologies and should allow all forms of UI and UI components to become accessible.